Skip to content

chore: update default mempool walk strategy #6059

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
wants to merge 11 commits into
base: develop
Choose a base branch
from

Conversation

obycode
Copy link
Contributor

@obycode obycode commented Apr 30, 2025

I think we have mined long enough with this setting to be confident to make it the default in the next release.

@obycode obycode requested a review from a team as a code owner April 30, 2025 20:49
@obycode obycode requested review from kantai, hstove and wileyj April 30, 2025 20:50
I think we have mined long enough with this setting to be confident to
make it the default in the next release.
@obycode obycode force-pushed the default-mempool-walk-strategy branch from 12d802a to 49d8974 Compare April 30, 2025 20:52
@hstove
Copy link
Contributor

hstove commented May 2, 2025

Overall looks good, but there are some test failures to be resolved first

@obycode
Copy link
Contributor Author

obycode commented May 2, 2025

Moving to draft until I get a chance to review the new test failures.

@obycode obycode marked this pull request as draft May 2, 2025 13:56
@obycode obycode added this to the 3.1.0.0.12 milestone May 27, 2025
@obycode obycode moved this to Status: 💻 In Progress in Stacks Core Eng May 27, 2025
Copy link

codecov bot commented May 28, 2025

Codecov Report

Attention: Patch coverage is 77.77778% with 56 lines in your changes missing coverage. Please review.

Project coverage is 77.32%. Comparing base (14230bf) to head (aa8ee60).
Report is 18 commits behind head on develop.

Files with missing lines Patch % Lines
testnet/stacks-node/src/tests/neon_integrations.rs 48.00% 39 Missing ⚠️
.../src/chainstate/stacks/tests/block_construction.rs 88.43% 17 Missing ⚠️

❗ There is a different number of reports uploaded between BASE (14230bf) and HEAD (aa8ee60). Click for more details.

HEAD has 38 uploads less than BASE
Flag BASE (14230bf) HEAD (aa8ee60)
150 112
Additional details and impacted files
@@             Coverage Diff             @@
##           develop    #6059      +/-   ##
===========================================
- Coverage    83.60%   77.32%   -6.28%     
===========================================
  Files          543      543              
  Lines       390781   390901     +120     
  Branches       323      323              
===========================================
- Hits        326694   302266   -24428     
- Misses       64079    88627   +24548     
  Partials         8        8              
Files with missing lines Coverage Δ
stackslib/src/config/mod.rs 67.71% <ø> (-5.62%) ⬇️
stackslib/src/core/mempool.rs 86.22% <100.00%> (-0.63%) ⬇️
stackslib/src/core/tests/mod.rs 71.98% <100.00%> (-19.51%) ⬇️
testnet/stacks-node/src/nakamoto_node/miner.rs 81.46% <100.00%> (-3.22%) ⬇️
.../src/chainstate/stacks/tests/block_construction.rs 82.40% <88.43%> (-15.31%) ⬇️
testnet/stacks-node/src/tests/neon_integrations.rs 51.80% <48.00%> (-13.20%) ⬇️

... and 224 files with indirect coverage changes


Continue to review full report in Codecov by Sentry.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 14230bf...aa8ee60. Read the comment docs.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

@aldur aldur assigned obycode and unassigned obycode May 28, 2025
@Jiloc Jiloc self-assigned this May 29, 2025
Comment on lines 622 to 628
// For NextNonceWithHighestFeeRate strategy, keep reset_nonce_cache = true
// to ensure considered_txs table is cleared for each block attempt
if self.config.miner.mempool_walk_strategy
!= MemPoolWalkStrategy::NextNonceWithHighestFeeRate
{
self.reset_nonce_cache = false;
}
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can you explain the need for this? I think this indicates a bug somewhere. It's important for performance to avoid clearing the nonce cache when possible.

Copy link
Contributor

@Jiloc Jiloc May 30, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

yep, I believe there was a bug in how we didn't clear the considered_txs caches between blocks. I refactored it a bit to be more performant, and avoid clearing the nonces when we can avoid it.

Anwyay the reason is that NextNonceWithHighestFeeRate uses a considered_txs table that tracks transactions evaluated during block construction to avoid duplicates or considering the same transaction multiple times. The problem is that considered_txs should be cleared between blocks (so previously rejected transactions can be reconsidered).

The bug was spot by the integration test clarity_cost_spend_down ( here the logs). When the miner was creating first block, it was going over all transactions, only getting the first 10 (because of the 5% tenure budget),added the others to the considered_txs cache. The cache wasn't cleared (because currently, the miner doesnt call the clear_mempool_caches when a block is mined), and the rest of the transactions were not mined.

@Jiloc Jiloc moved this from Status: 💻 In Progress to Status: In Review in Stacks Core Eng May 30, 2025
@Jiloc Jiloc marked this pull request as ready for review May 30, 2025 16:03
@Jiloc Jiloc requested a review from a team as a code owner May 30, 2025 16:03
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
Status: Status: In Review
Development

Successfully merging this pull request may close these issues.

3 participants